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The  main  focus  of  this  study  is  on  issues  related  to  the  effect  of 
different  instructional  methods  on  the  consistency  and  types  of 
algorithms  developed  by  students  as  well  as  on  the  underlying 
dimensionality  of  the  test. 

The  first  phase  of  the  experiment,  in  which  the  dimensionality  of 
an  achievement  test  across  different  learning  stages  under  two 
different  instructional  methods  was  investigated,  was  carried  out  in 
January,  1980,  using  about  160  seventh  graders  at  a  junior  high  school. 
The  motivation  for  the  experiment  originated  with  the  earlier  studies  of 
Tatsuoka  &  Birenbaum  (1979,  1981).  Unidimensionality  of  a  test  is 
essential,  at  present,  whenever  latent  trait  models  are  to  be  involved 
in  order  to  measure  an  individual's  location  on  an  ability  or 
achievement  continuum  underlying  the  test.  The  study  demonstrated  that 
instructional  methods  affect  the  dimensionality  of  test  scores  to  a 
large  extent.  As  to  the  stability  of  the  responses  across  parallel 
items,  the  results  indicate  that  in  the  early  stages  of  learning, 
students  tend  to  change  their  rules  of  operation  during  the  test.  This 
phenomena  causes  a  clear  violation  of  the  local  independence  assumption 
which  is  one  of  the  most  essential  assumptions  of  the  Item  Response 
theory  (or  latent  trait  theory). 

The  results  of  this  study  underscore  the  need  for  combining  the 
efforts  of  cognitive  psychology  and  psychometrics,  as  Glaser  mentioned 
in  his  1981  paper. 

The  authors  summarize  empirical  results  of  error  analysis  and 
factorial  studies  based  on  the  first  phase  of  experimental  data  in  this 
study.  A  further  completed  study  will  be  reported  in  a  future  research 
report. 

Methodology 

The  Data  Collection  Procedures 

The  dataset  used  in  this  study  (to  be  referred  hereafter  as  the 
"January  data")  was  collected  in  the  winter  of  1980.  Two  methods  for 
teaching  the  concept  of  signed  numbers,  one  entitled:  "The  Postman 
Stories  "  and  the  other  "The  Number-Line  Method  "  were  Implemented  on 
the  PLATO  system.  157  seventh  graders  from  a  Junior  High  School  were 
randomly  assigned  to  study  the  topic  under  one  of  those  methods  without 
any  assistance  from  teachers.  Each  Instructional  unit  was  divided  into 
two  sessions;  one  for  teaching  addition  and  the  other  for  teaching 
subtraction.  After  addition  was  Introduced  and  practiced  (including  a 
check-up  with  feedback)  a  test  consisting  of  52  free  response  items 
representing  13  tasks  of  4  parallel  items  each  was  administered  on  the 
PLATO  system.  A  copy  of  this  test  is  presented  in  Appendix  1.  Eighty- 
one  students  from  both  experimental  groups  completed  that  test  twice. 
Their  responses  constitute  the  four  datasets  used  in  the  current  study. 
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The  Instructional  Methods 


The  Postman  Stories,  This  method  for  teaching  signed  numbers  is 
part  of  the  Madison  Mathematics  Project  (Davis,  1964).  This  method 
associates  positive  and  negative  integers  with  checks  and  bills, 
respectively.  Addition  is  represented  by  a  postman  bringing  something 
(a  check  or  a  bill),  while  subtraction  corresponds  to  the  postman's 
taking  something  from  the  home. 

In  the  programmed  version  of  this  lesson  which  was  used  in  the 
January  experiment  addition  was  presented  by  six  types  of  tasks.  The 
first  one  of  adding  two  positive  integers  appeared  on  the  screen  both  as 
a  number  story:  M+2  +  +5"  and  as  a  Postman  Story:  "On  Monday,  the  postman 
brings  you  a  {check]  for  $5  and  he  (brings)  a  [check]  of  $2.  You  are  Irlcherl  by 
$7.  The  other  five  tasks  appeared  on  the  screen  either  as  a  postman 
story  to  which  the  student  had  to  fill  in  the  number  story  (i.e.  the 
equation)  or  vice  versa. 

Subtraction  was  presented  in  a  simmilar  way.  A  problem  of 
subtracting  two  positive  integers  appeared  on  the  screen  both  as  a 
postman  and  a  number  story  as  follows:  "+5  -  +2  »  +3”,  "On  Monday  the 
postmans  brings  you  a  [check!  for  $5  and  he  [takes  awayl  a  [check!  for  $2.  You 
are lricher]  by  $3."  For  the  other  five  tasks  in  subtraction  the  student 
had  to  fill  in  either  the  number  story  or  the  postman  ones.  Table  1 
presents  the  structure  of  the  problems  that  were  solved  during  the 
lesson.  Each  problem  had  to  be  solved  correctly  before  the  student  could 
preceed  to  the  next  one.  Whenever  a  mistake  was  made  the  screen  would 
flash  in  that  part  of  the  problem  and  a  short  explanation  of  the 
situation  which  was  presented  in  the  problem  would  appear  before  the 
student  was  asked  to  try  it  again. 

The  Number-Line  method.  This  geometric  method  for  teaching  signed 
numbers  introduces  the  concept  of  negative  integers  by  means  of  moving  a 
pointer  to  the  left  of  zero  on  the  number  line.  This  unit  was  originally 
implemented  on  PLATO  by  T.  Weaver.  In  that  version  of  the  lesson,  rules 
were  given  as  to  which  direction  to  move  the  pointer  for  addition  and 
subtraction  of  integers.  These  rules  were  deleted  from  the  lesson  in  the 
January  experiment  in  order  to  make  it  comparable  to  the  Postman  lesson. 
The  lesson  begins  with  an  introduction  of  the  number  line  and  the 
location  of  positive  and  negative  Integers  on  it.  The  students  practice 
locating  numbers  on  the  number  line  by  moving  a  pointer.  After  getting 
acquainted  with  the  number  line,  the  students  are  asked  to  solve  an 
addition  problem  of  two  whole  numbers:  5  +  2  by  using  the  pointer.  Five 
more  addition  problems,  consisting  of  different  combinations  of  positive 
and  negative  integers,  are  presented.  The  students  are  expected  to 
generate  the  rules  as  to  which  direction  to  move  the  pointer  in  order  to 
solve  them.  (The  problems  are  Identical  to  those  presented  in  the 
''Postman11  lesson.  For  the  structure  of  the  problems  see  Table  1.) 


The  two  computerized  instructional  units,  the  Postman  stories  and 
the  Number  Line  method,  as  used  in  the  January  experiment,  were  meant  to 
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Table  1 


The  types  of  problems  presented  in  the  computerized  version  of  the 
"Postman  Stories"  and  the  "Number-Line"  instructional  methods. 


addition 


subtraction 


.  +L  +  +S 

2.  +L  +  -S 

3.  +S  +  -L 

4.  -L  +  -S 
3.  -S  +  -L 
6.  -S  +  -L 


+L  -  +S 
+L  -  -S 
+S  -  -L 
-L  -  +S 
-S  -  +L 
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teach  the  concept  of  signed  numbers  in  two  different  representations. 
Both  of  these  approaches  deliberately  refrained  from  stating  rules  for 
solving  problems,  in  contrast  to  the  teachers'  method,  which  gave  rise 
to  data  collected  in  November,  1979  (see  Birenbaum  &  Tatsuoka,  1980). 

The  difference  between  the  November  and  the  January  teaching  methods  in 
this  respect  is  simlar  to  the  one  made  by  Winograd  (1980)  concerning  the 
pre-AI  programs  for  language  and  the  basic  AI  model.  Winograd 
emphasizes  that  the  critical  element  in  the  AI  basic  model  is  the 
explicit  manipulation  of  a  formal  representation.  In  this  model  as 
Winograd  states:  "Operation  carried  out  on  the  representaion  structures 
are  justified  not  by  facts  about  language,  but  by  the  correspondence 
between  the  representation  and  the  world  being  described.  This  is  the 
sense  in  which  such  programs  were  said  to  "understand"  the  words  and 
sentences  they  dealt  with  where  the  earlier  machine  translation  programs 
had  manipulated  them  without  understanding"  (ibid  p.211).  Both  of  the 
teaching  units  had  identical  check-up  problems,  one  set  after  addition 
and  the  other  after  subtraction.  These  check-ups  consisted  of  items 
parallel  to  the  items  the  student  practiced  solving  in  the  lesson.  The 
correct  answers  were  provided  during  the  lesson  and  the  check-ups. 

The  data  analysis  procedures.  The  data  analysis  procedures  used  in 
this  study  are  similar  to  those  described  in  Birenbaum  &  Tatsuoka  (1980) 
for  the  November  data.  Figure  1  summarizes  the  design  of  the  data 
collection  and  analyses  procedures. 

The  Effect  of  the  Point  of  Time  in  the  Learning  Process  and  the  Type  of 
Instruction  on  the  Consistency  of  Responses  Across  Parrallel  Items 

Consistent  responses  across  parallel  items  can  be  considered  as  an 
indicator  of  the  student's  level  of  "mastery"  or  confidence  in  his 
hypothesized  rule.  Psycholinguistics  explains  language  acquisition  by 
children  as  a  process  of  hypothesis  generating  and  then  testing  them 
against  the  language  spoken  around  them  (see  for  example  Moskowitz, 
1978).  Similarly,  when  a  student  is  faced  with  a  problem  solving  task 
in  mathematics,  he  will  generate  an  hypothesis  (e.g.,  rule)  and  test  it 
on  different  problem  types.  The  rule  will  be  changed  or  modified  only 
when  the  student  recognizes  "new  circumstances"  under  which  his 
hypothesis  can  no  longer  be  confirmed.  As  stated  by  Davis  et.  al. 

(1979)  concerning  this  matter:  "...discriminations  develop  only  as 
finely  as  they  are  needed..."  (p.116).  Thus,  when  the  student  is  still 
in  a  stage  of  testing  and  modifying  his  hypothesis,  l.e.,  when  mastery, 
even  for  incorrect  rules  has  not  yet  been  achieved,  one  is  likely  to 
find  less  consistent  responses  across  parallel  subtests.  This  is  due  to 
the  fact  that  each  new  task  provides  information  which  either  confirms 
the  hypothesis  or  disconfirms  it,  in  which  case  the  hypothesis  is 
modified  and  tested  on  the  next  task  and  modified  again  if  necessary  as 
the  next  task  is  approached. 

We'll  illustrate  this  process  with  examples  from  interviews  with 
two  students  who  were  taking  the  PLATO  lessons  on  signed  numbers. 


IS?  SEVENTH 


Figure  1.  The  Research  Design 
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Case  1 .  The  response  pattern  presented  in  Table  2  for  the  sign- 
numbers  test  was  given  by  a  7th  grade  student  who  studied  the  "Postman 
Stories"  and  was  interviewed  while  taking  the  lesson  and  the  test. 
During  the  test  there  was  no  feed-back  or  judgement  concerning  the 
correct  answer. 

The  coded  consistency  pattern  for  this  student  is  as  follows.  The 
explanation  of  the  terminology  is  given  in  Birenbaum  &  Tatsuoka 
(1980). 


Task 

Task 

Consistency 

no. 

type 

code 

4 

S-(-L) 

0 

2 

-S-L 

0 

13 

-S-+L 

4 

1 

-S-(-L) 

3 

8 

-L-(-S) 

3 

7 

L-S 

1 

16 

S-L 

1 

10 

-L4-S 

1 

5 

-S+L 

1 

15 

-L+S 

1 

3 

L+— S 

1 

11 

S+-L 

1 

This  response  pattern  indicates  quite  a  high  degree  of  stable 
answers  to  most  of  the  test  items.  At  least  on  three  parallel  subtests 
the  student  is  using  the  same  strategy  for  solving  parallel  i terns , 
(excluding  tasks  4  and  2).  It  also  indicates  that  the  student  has 
mastered  the  correct  algorithm  for  addition  of  signed  numbers  since  he 
consistently  answers  all  types  of  addition  problems  correctly. 

As  can  be  inferred  from  this  consistency  pattern,  the  rule  for 
subraction  that  this  student  has  mastered  is  as  follows:  Always  change 
negative  integers  to  positive  ones  and  add  the  two  integers.  The 
"motivation"  behind  this  rule  is  expressed  by  the  student  during  the 
interview*  as  follows:  "When  there  are  two  minuses  next  to  each  other 
you  change  them  to  plus,"  To  the  interviewers  question  "What  do  you 
mean  by  'next  to  each  other?'"  the  student  answers:  "Next  to  each  other 
is  either:  -(]-  or  [] — •" 

Comparing  the  student's  response-patterns  on  the  four  parallel 
subtests  provided  more  insight  into  the  process  that  eventually  led  him 
to  generating  this  rule.  On  the  first  subtest  the  student  doesn't 
distinguish  between  addition  and  subtraction.  He  applies  the  rule  for 
addition  to  the  first  five  subtraction  problems  (i.e.,  treats  a  minus 
operation  as  if  it  were  a  plus  operation).  It  is  unclear  what  he  is 
doing  on  the  6th  subtraction  problem  since  he  typed  the  number  11  as  the 
answer  to  the  problem:  13  -  (-4).  However,  on  the  7th  subtraction 
problem,  in  which  for  the  first  time  in  the  test  he  is  faced  with  a 


Given  by  the  First  Student  During  the 
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subtraction  problem  that  has  an  explicit  plus  sign  for  one  of  the 
integers,  i.e.,  -3  -  +12,  he  suddenly  realizes  that  something  is  wrong 
with  his  former  rule.  Thus,  from  here  on  he  switches  to  his  new  rule 
concerning  the  two  minuses  being  close  to  each  other.  He  extends  his 
rule  to  the  case  of  three  minuses  (problem  17)  which  according  to  his 
conception  has  two  cases  of  two  minuses  close  to  each  other.  In  his 
responses  on  the  second  subtest  it  can  be  noticed  that  the  "new"  and  the 
"old"  rules  are  still  competing.  Thus,  the  problems  in  tasks  2  and  4 
are  still  answered  using  the  "old"  rule  (see  responses  to  items  18  and 
20).  (This  causes  the  two  codes  of  zero  in  the  consistency  pattern  for 
tasks  2  and  4).  However,  on  the  next  two  subtest  the  "new"  rule 
stabilizes  and  is  being  used  throughout  the  rest  of  the  subtraction 
problems  (except  for  tasks  7  and  16  which  are  treated  as  a  different 
category,  as  stated  by  the  student  in  the  interview:  "'cause  it  is  a 
regular  case  of  two  positive  numbers." 

Case  2.  The  second  example  of  hypothesizing  a  rule,  testing  it  and 
switching  to  another  rule  when  the  first  one  breaks  down  is  illustrated 
below  using  data  from  another  interview.  This  student  was  interviewed 
while  she  was  studying  the  "Number-Line"  lesson  and  the  part  of  the 
protocol  we  are  referring  to  deals  with  the  practice  problems  that  were 
given  at  the  end  of  the  subtraction  unit.  At  this  point  feedback 
regarding  the  correct  answers  was  provided. 

The  first  inductive  rule  which  this  student  generates  is  as 
follows:  "on  the  right  side  of  the  number  line  you  go  to  the  right  to 
subtract  and  to  the  left  to  add.  On  the  left  side  of  the  number  line  you 
go  left  to  subtract  and  right  to  add"^%  This  incorrect  rule  yields 
correct  answers  to  problems  such  as:  4  -  (-3)  =  7;  1  -  (-3)  =  4;-2-5=i- 

-4  -  2  =■  -6  and  the  student  gets  more  and  more  self  assured 

about  her  rule  as  can  be  noticed  in  her  remarks:  "Ohh,  I'm  right;  I  was 
right!!"  (Ibid  p.  10). 

However,  this  rule  breaks  down  with  problems  involving  two  numbers 
with  the  same  signs  such  as:  -4  -  (-3).  Following  her  rule  the  student 

gets  an  answer  of  -7.  She  seems  to  be  surprised  to  see  the  message  that 

her  answer  is  wrong  and  after  practicing  some  more  problems  she  induces 
the  following  new  rule:  if  there  is  an  explicit  plus  sign  in  the 
problem,  add  and  put  the  sign  of  the  first  number.  Or,  as  stated  by  the 
student  when  explaining  the  way  she  solves  the  problem  -1  -  (+2):  "I  added 
because  it  says  one  plus  two.  Now  I  get  it!  One  minus  plus  two.  O.K., 

I  don't  know  why  these  things  are  here.  It  says  one  and  then  in  the 
parentheses  it  says  plus  two,  and  this  sign  right  here  [the  sign  in 
front  of  the  first  number]  you  are  supposed  to  put  on  the  answer."  As 
can  be  seen  from  this  example  and  form  the  previous  one,  in  the  stage  of 
learning,  when  mastery  has  not  yet  been  reached,  students  are 
hypothesizing  their  algorithms  and  testing  them.  The  change  or 
modification  of  an  algorithm  occurs  only  when  the  previous  rule  seems  to 
break  down.  Thus,  any  kind  of  feedback,  either  of  the  form  of  judging 
the  answer  and  providing  the  correct  one  or  even  the  next  type  of  task 
in  the  test  serves  as  a  reference  against  which  the  student  is  testing 
his  hypothesized  algorithm. 
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It  should  be  noticed  that  if  indeed  a  test  serves  sometimes  as  a 
learning  experience  causing  students  to  switch  strategies  on  the  basis 
of  clues  they  got  from  the  different  tasks  or  types  of  questions,  then 
applying  latent  trait  models  to  such  tests  is  inappropriate.  It  is  a 
clear  violation  of  the  local  independence  assumption  which  is  one  of  the 
most  critical  assumptions  of  ICC  theory  (see  Lord  &  Novick  1968). 

The  following  is  one  of  the  equations  given  by  Lord  &  Novick 
(1968)  as  a  definition  of  "local  independence"; 
hi(yi*i  ©;  y2*»y3*»***»yn*>  =  fi(yi*i  ©)  (eq.16.3.2  p.36i) 

This  shows  that  under  local  independence,  the  conditional  distribution 
hi  of  an  item  score  y j*  for  fixed  values  of  0,  y2* ,y3* » • • *yn*  doesn't 
depend  on  other  item  scores  y2*>y3*>»  ..yn.  Thus,  for  respondents  with 
the  same  0  value,  any  association  between  the  items  should  disappear. 

In  pointing  out  different  kinds  of  arguments  to  defend  the 
introduction  of  the  local  independence  requirement,  Mokken  (1971) 
states:  "In  the  first  place  there  are  some  general  substantive 
considerations  concerning  proper  measurement  which  may  give  rise  to  this 
principle.  What  these  considerations  amount  to  is  that  when  we  have 
determined  the  value  of  a  subject  and  kept  it  constant,  all  systematic 
variation  in  the  outcomes  of  measurement  of  that  value  should  disappear, 
leaving  only  purely  random  variation"  (ibid  p.76).  This  of  course  is 
assuming  that  we  are  measuring  a  stable  trait.  However,  as  was  shown  in 
section  of  this  report,  when  the  test  is  given  during  the  stage  of 
learning,  when  the  algorithms  haven't  yet  been  stabilized,  but  are 
rather  being  modified  during  the  test,  we  can't  talk  of  a  stable  latent 
trait.  At  most  we  consider  a  latent  state  which  is  likely  to  change 
during  the  testing  situation.  Thus,  the  local  independence  assumption 
which  implies  that  the  order  of  the  test  items  should  not  affect  the 
estimates  of  the  examinees'  ability  level  is  clearly  violated  in  the 
case  where  the  order  of  the  test  items  does  make  a  difference.  In  such  a 
case  there  exists  a  systematic  variation  which  is  attributed  to  another 
related  dimension  -  another  latent  state  which  reflects  another 
generated  algorithm.  Therefore,  any  measurment  model,  including  the  ICC 
family,  that  assumes  a  static  rather  than  a  dynamic  trend  in  the 
responses  on  a  given  test  would  be  inappropriate  for  use  in  those  stages 
in  the  instructional  process  in  which  the  algorithms  are  not  yet 
stabilized • 

A  Comparison  of  the  Consistency  of  Responses  in  the  Two  Experimental 
Groups  ~ ~  _ 

In  order  to  compare  the  consistency  of  responses  across  parallel 
items  in  the  two  experimental  groups,  a  discriminant  analysis  was 
performed.  A  score  of  1  was  assigned  whenever  at  least  3  out  of  4 
parallel  items  got  the  same  code  (1,  2,  3,  or  4)  otherwise  a  score  of 
zero  was  assigned. 

The  results  of  this  anlysis  for  the  first  test  (the  one  that  was 
given  at  the  end  of  the  addition  lesson)  are  presented  in  Table  3.  As 
can  be  seen  in  the  table,  there  is  an  overall  significant  difference 
between  the  two  experimental  groups  with  respect  to  the  consistency  of 
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Table  3 


Means,  Canonical  Correlation  and  Standardized 
Canonical  Discriminant  Function  Coefficients  of  Consistency 
for  the  Two  Treatment  Groups  in  the  First  Test 


group _ N _ Means 

"Postman*’  1  39  -.49 

"Number  Line"  2  42  .45 


X  18 

Rc  .43 

(1-A)100  18.40 

p<  .01 


Standardized  Function  Coefficients 


Tas  k 

Task 

No. 

Type 

2 

-S-L 

.61 

13 

-S-+L 

-.46 

7 

L-S 

.54 

16 

S-L 

-.68 

10 

-L+-S 

-.37 

responses.  The  dimension  along  which  the  two  groups  differ  involves 
four  of  the  subtraction  tasks  and  one  addition  task.  The  group  with  the 
higher  Centroid  (group  1)  is  characterized  as  having  lower  consistency 
scores  on  tasks;  13)  -S-+L;  10)  -L+- S;  16)  S-L  and  higher  consistency 

scores  on  tasks  2)  -S-L  and  7)  L-S.  Appendix  2  presents  the  percent  of 
responses  for  every  task  on  each  of  the  five  coded  categories  (0  -  4) 
where  0  indicates  inconsistent  responses  across  parallel  items  and  codes 
1-4  indicate  that  at  least  3  out  of  4  responses  on  parallel  items  fell 
into  this  coded  category.  As  can  be  seen  in  this  table*  41%  of  the 
students  in  group  2  gave  inconsistent  responses  on  task  13  compared  to 
21%  in  group  1.  The  most  common  error  in  the  latter  group  is  the  one 
coded  as  E2.  54%  of  the  students  in  group  1  consistently  responded  in 

this  category  as  compared  to  33%  in  group  2.  In  task  10*  43%  of  the 
students  in  group  2  gave  inconsistent  responses  compared  to  28%  in  group 
1.  46%  of  the  students  in  group  1  answered  the  items  in  this  task 

correctly,  as  compared  to  26%  in  group  2.  In  task  16  the  rate  of 
inconsistent  responses  in  group  1  was  18%  whereas  the  inconsistency  rate 
for  the  other  group  was  twice  as  much  (36%).  In  group  1*  56%  of  the 
students  consistently  answered  task  16  items  correctly  as  compared  to 
45%  in  group  2. 

In  task  2*  however,  41%  of  the  students  in  group  1  gave 
inconsistent  responses  compared  to  24%  in  group  2.  Only  10%  in  group  1 
and  21%  in  group  2  consistently  answered  correctly  items  in  this  task. 
The  most  common  consistent  error  was  E3.  In  task  7  21%  of  the  students 
in  group  1  as  compared  to  12%  in  group  2  gave  inconsistent  reponses.  56% 
in  group  J  and  45%  in  group  2  is  the  rate  of  correct  answers  in  this 
task.  The  most  common  error  was  E4:  38%( ! )  of  the  students  in  group  2 

and  18%  of  the  students  in  group  1  consistently  committed  this  error 
(e.g.,  getting  an  answer  of  -2  to  the  problem:  8-6). 

Since  the  two  experimental  groups  were  exposed  to  exactly  the  same 
tasks  throughout  their  entire  learning  experience,  the  significant 
differences  in  the  consistency  rate  between  the  two  groups  seems  to 
imply  that  different  information  processings  are  involved.  In  the 
second  test  however,  the  overall  difference  in  consistency  was  not 
significant  (p<.07).  Table  4  presents  the  consistency  rates  in  each  of 
the  two  tests  for  each  of  the  experimental  groups.  As  can  be  seen  in 
the  table,  the  highest  consistency  rate  was  given  in  the  first  test  by 
group  1.  At  least  50%  of  the  tasks  in  that  test  were  systematically 
answered  by  84%  of  the  students  in  group  1  compared  to  65%  of  the 
students  in  group  2.  It  should  be  noticed  that  this  rate  of  consistency 
is  far  lower  than  the  one  found  in  the  November  data.  In  that  case,  the 
test  was  given  after  a  three-week  period  of  intensive  classroom 
instruction  Including  practice  exercises,  quizzes  and  tests  on  this 
topic.  At  that  point  the  students  already  had  pretty  stable  or  fixed 
algorithms  and  that  was  reflected  in  the  high  rate  of  consistent 
responses  in  the  test.  92%  of  the  students  gave  systematic  answers  on 
13  -  16  test  tasks  (excluding  those  students  who  consistently  answered 
correctly).  These  figures  clearly  indicate  that  the  stage  in  the 
instructional  process,  in  tdiich  the  test  is  given,  does  make  a 
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Table  4 


Consistency  Rate:  Percentage  of  Consistent  Responses 
to  12  Test  Tasks. 


No.  of  Tasks  \ _ Pusijaaj _ Number.  Line  — 

consistently  answered  \  T1 _ T2 _ T1 _ T2 


n 

(12) 

15 

11 

11 

12 

n-1 

(ID 

7 

17 

10 

14 

n-2 

(10) 

22 

8 

15 

7 

n-3 

(9) 

17 

13 

16 

16 

n-4 

(S) 

14 

15 

5 

11 

n-5 

(7) 

9 

11 

8 

7 

n-6 

(6) 

5 

7 

14 

14 

n-7 

(5) 

5 

— 

10 

3 

n-3 

(4) 

1 

8 

3 

2 

n-9 

(3) 

— 

2 

3 

7 

n-10 

(2) 

— 

4 

3 

2 

n-11  (1) 


5 


4 


2 


5 
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difference,  as  far  as  the  consistency  of  the  responses  across 
parallel  item  is  concerned. 

Are  The  Algorithms  Developed  by  Students,  Following  Two  Different 
instructional  MetWJs, The Same? 

The  two  teaching  methods  used  in  the  January  experiment  differ  from 
each  other  with  respect  to  the  type  of  representation  they  offer  for 
teaching  the  concept  of  signed  numbers.  The  "Number-Line"  method  uses  a 
representation  from  the  geometric  domain  whereas  the  "Postman  Stories" 
method  relies  on  its  representation  of  a  common  bookkeeping  experience. 
Both  methods  as  presented  in  the  January  experiment  deliberately 
refrained  from  stating  explicit  rules  for  solving  the  problems. 

Students  were  expected  to  develop  their  own  algorithms  based  on  the 
representation.  Thus,  it  becomes  interesting  to  compare  the  algorithms 
constructed  following  each  of  these  instructional  methods.  The 
following  is  a  complete  list  of  the  algorithms  identified  from  the 
students'  response  patterns  in  the  two  experimental  groups.  The 
algorithms  are  divided  into  three  categories:  those  that  were  common  to 
the  two  instructional  methods  and  those  that  were  unique  to  each  method. 
The  algorithms  for  subtraction  and  addition  are  described  separately 
since  the  majority  of  the  students  distinguished  between  addition  and 
subtraction  operations  and  used  different  rules  for  each  operation. 
(Appendix  3  presents  the  response-patterns  and  the  number  of  students 
whose  answers  match  those  patterns.) 


A)  Algorithms  identified  in  common  response-patterns  for  two 
instructional  styles. 

^ubtra£tioii:^  1st.  ^test_(Tl ) 

Please  note  that  the  numbers  refer  to  the  response  patterns 
presented  in  Appendix  3.  N  «  Number  line,  and  P  -  Postman  stories. 

N1,P1)3  Subtracts  |L|  -  |S|.  Puts  the  sign  of  the  number 
with  the  larger  absolute  value  in  the  result. 

N4,P5)  Subtracts  |L|  -  |Sj.  Puts  the  sign  which  is  attached 
to  the  second  number  in  the  result. 

Nl,Pl)  Subtracts  |L|  -  |S|.  Puts  a  minus  sign  in  the  result. 
N5,P7)  Uses  the  rule  for  addition  after  changing  the 
operation  sign  from  a  minus  to  a  plus.  (Except  for  tasks 
7  and  16). 

N6,P9)  Same  as  N5,P7. 

N2,P4)  When  there  is  an  explicit  plus  sign  in  the  problem 
adds  and  puts  a  plus  sign  in  the  result.  When  the  only 
explicit  signs  are  minuses,  subtracts  and  puts  a  minus  sign 
in  the  result. 

Subtraction 2nd. 

n/7p7)T*  T ids  the~a1>8<>lut^  values.  Puts  a  minus  sign  in  the 
result  if  at  least  one  number  has  a  minus  sign.  Otherwise 
puts  a  plus  sign. 
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Subtraction:^  T1A  T2_ 

N1  l(T2);Pl3(Tl7:  Subtracts  /L|  -  |S|#  Puts  a  plus  sign  in 
the  result. 

Addition:  __  T1 

N(C);P(B):  When  a  minus  sign  is  attached  to  the  second 
number,  subtracts  |L|  -  |S|.  Otherwise,  adds  the  two 
absolute  values.  Puts  a  plus  sign  in  the  result. 

Addiction:  _T2 

N(C)7p7d7:~  Subtracts  |L|  -  |S|.  Puts  the  sign  of  the 
number  with  the  larger  absolute  value  in  the  result. 

N(A);P(G):  Subtracts  J  T.  \  -  jS|.  Puts  a  plus  sign  in  the 
result. 

Addi_tij>n^  _T1_,_T2 

N(D,T2);P(A,T1):  Subtracts  |L|  -  |S|.  Puts  a  minus  sign 
in  the  result. 

B)  Algorithms  identified  in  unique  response-patterns  for  the 
number  line  method 


S  £.a£  „T1_ 

T)  If  the  first  number  is  to  the  left  of  the  second  one  on 
the  number  line,  counts  to  the  right  the  number  of  steps 
indicated  by  the  second  number. 

If  the  first  number  is  to  the  right  of  the  second  one, 
counts  to  the  left,  the  number  of  steps  indicated  by  the 
second  number. 

11)  The  sign  of  the  first  number  determines  which  direction  to  jump  on 
the  number  line.  When  the  first  number  is  negative,  jumps  to  the  right 
on  the  number  line,  the  number  of  steps  indicated  by  the  second  number. 
8)  The  operation  sign  detemlnes  the  sign  of  the  result. 

The  sign  of  the  second  number  determines  the  operation  to 
be  taken. 

3)  Subtracts  |L|  -  |S|.  Puts  the  sign  which  is  attached 

to  the  number  with  the  largest  absolute  value  in  the  result. 

12)  Adds  the  absolute  values.  Puts  a  minus  sign  in  the 
result  (except  for  task  7  in  which  subtracts  1L|  -  |S|). 

13)  Subtracts  |L|  -  |S|.  Puts  a  minus  sign  the  the  result 
(except  for  task  7  and  16). 

14)  Subtracts  |L|  -  |S|.  Puts  a  plus  sign  in  the  result 
(except  for  task  7  for  which  puts  a  +  sign). 

Subtraction  T2j. 

2)~  ?inds  The  distance  between  the  two  numbers  on  the 
number-line.  Puts  a  minus  sign  in  the  result  (except  for 
task  7). 

4)  Jumps  to  the  left  of  the  first  number  on  the  number  line 
the  number  of  steps  Indicated  by  the  second  number. 

12)  When  the  first  number  is  smaller  in  absolute  value  than 
the  second,  adds  and  puts  a  plus  sign  in  the  result.  When 
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the  first  number  is  the  larger  in  absolute  value,  subtracts 
and  puts  a  minus  sign  in  the  result. 

10)  When  the  two  numbers  are  positive  or  when  there  is  at 
least  one  explicit  plus  sign  in  the  problem,  subtracts  and 
puts  a  plus  sign  in  the  result.  Otherwise,  adds  and  puts 
a  minus  sign. 

13)  The  sign  attached  to  the  number  with  the  larger 
absolute  value  determines  the  operation  to  be  taken.  Puts 
a  plus  sign  in  the  result  (except  for  task  7). 

Addict  ionj^  jn_ 

B)  The  sign  of  the  first  number  determines  the  operation  to 
be  taken. 

E)  Subtracts  |Lj  -  |S| .  Puts  the  sign  which  is  in  front  of 
the  second  number  in  the  result. 

D)  Adds  the  absolute  values  and  puts  the  sign  of  the  first 
number  in  the  result. 


Addition^  JT2^ 

B)  When  the  first  number  is  smaller  in  absolute  value  than 
the  second,  adds  and  puts  a  plus  sign  in  the  result.  When 
the  first  number  is  larger  in  absolute  value,  subtracts  and 
puts  a  minus  sign  in  the  result. 

F)  Jumps  to  the  right  of  the  first  number  on  the  number  line 


C)  Algorithms  Identified  in  Unique  Response-Patterns  fo 
the  Postman  Stories  Method. 


Sub tractionj_  _T1__ 

3)  Applies  the  addition  algorithm  to  the  subtraction 
problems.  I.e.,  when  the  two  number  signs  are  the  same, 
adds  the  absolute  values.  When  the  are  different,  subtracts 
|L|  -  | S i •  Puts  the  sign  of  the  number  with  the  larger 
absolute  value  in  the  result. 

10)  Subtracts  |L|  -  |S|.  When  the  first  number  is  the 
larger  in  absolute  value,  puts  a  plus  sign  in  the  result. 
When  the  first  number  is  the  smaller  in  absolute  value, 
puts  a  minus  sign. 

12)  When  the  first  number  is  the  larger  in  absolute  value, 
subtracts  |L|  -  |S|.  When  the  first  number  is  the  smaller 
in  absolute  value,  adds.  Puts  the  sign  of  the  number  with 
the  larger  aboslute  value  in  the  result. 


Subt ra£ t ion JT2_ 

1)"*  When  the  first  number  is  positive,  adds  and  puts  a  plus 
sign  in  the  result.  When  the  first  number  is  negative, 
subtracts  and  puts  a  minus  sign  in  the  result. 

6)  Subtracts  |L|  -  |S|.  Puts  the  sign  which  is  attached 
to  the  number  with  the  larger  aboslute  value  in  the  result 
(except  for  task  16). 

Addition^  _T1_ 

TT)"*  When^there  is  an  explicit  plus  sign  attached  to  the 
second  number  adds  and  puts  a  plus  sign  in  the  result. 
Otherwise  subtracts  and  puts  a  minus  sign. 
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H)  Adds  the  two  absolute  values.  When  there  is  at  least 
one  negative  number,  puts  a  minus  sign  in  the  result. 

Otherwise,  puts  a  plus  sign. 

F)  Adds  the  two  absolute  values.  Puts  a  minus  signs  in  the 
result. 

I)  Subtracts  |L|  -  |S|.  Puts  a  plus  sign  in  the  result. 

G)  Subtracts  |L|  -  |S|.  Puts  a  minus  sign  in  the  result. 

Addiction:  _JT2 

A)  When  the  first  number  is  positive,  adds  and  puts  a 
plus  sign  in  the  result.  When  the  first  number  is  negative, 
subtracts  and  puts  a  minus  sign  in  the  result. 

F)  When  the  first  number  is  the  smaller  in  absolute  value 
adds.  Otherwise  subtracts  |L|  -  |S|.  Puts  the  sign  of  the 
first  number  in  the  result. 

B)  Adds  the  two  absolute  values.  Puts  the  sign  of  the 
first  number  in  the  result  (except  for  task  16). 

C)  Adds  the  two  absolute  values.  Puts  a  plus  sign  in  the 
result. 

I)  When  the  first  number  has  no  explicit  number  sign 
attached  to  it,  adds  the  two  absolute  values  and  puts  a 
plus  sign.  Otherwise  answers  correctly. 

The  response  patterns  that  were  common  to  both  experimental  groups 
seem  to  bear  no  clear  relation  to  either  one  of  the  representations  that 
were  used  during  the  instructional  process.  The  algorithms  identified 
from  these  responses  can  be  explained  as  reflecting  a  mental  process 
that  refers  to  the  syntactic  attributes  of  the  problem  rather  than  to 
the  semantic  ones.  Factors  such  as  the  sign  of  the  first  number,  the 
location  of  the  number  with  the  larger  absolute  value,  etc.  seem  to  play 
an  important  role  in  determining  whether  to  add  or  subtract  the  numbers 
and  which  sign  to  choose  for  the  result.  Some  responses  reflect  no 
distinction  between  subtraction  and  addition  operations.  Others 
indicate  a  lack  of  distinction  between  Integers  with  different  signs  and 
integers  with  the  same  signs. 

Among  the  algorithms  that  where  identified  from  unique  patterns  in 
the  number-line  data,  there  are  some  that  are  closely  related  to  the 
representation  used  throughtout  the  instruction.  Algorithms  such  as  7, 
11(T1);  2,  4;  F(T2)  clearly  indicate  the  use  of  a  number  line  for 
deriving  the  answer.  As  to  the  unique  algorithms  for  the  postman 
stories,  there  was  no  clear  indication  that  the  representation  was  used. 

It  is  Interesting  to  note  that  even  though  the  total  test  score  was 
not  significantly  different  between  the  two  experimental  groups  the 
number  of  different  algorithms  for  the  subtraction  problems  in  the  group 
who  studied  the  Number-Line  method  was  significantly  larger  than  that  of 
the  group  which  studied  the  Postman  Stories.  This  again  points  to  the 
Important  gain  in  information  provided  by  error  analysis  as  compared  to 
the  conventional  scoring  system. 
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The  Postman-Stories  unit  presented  a  clear  correspondence  between 
the  elements  of  the  problems  and  their  representation,  i.e.,  bring/ take 
away  for  operations  and  checks/bills  for  number  signs*  The  Number-Line 
method  didn't  make  a  clear  distinction  between  the  operation  signs  and 
the  number  signs  in  its  representation*  The  fact  that  students  tended 
to  use  more  the  representation  of  the  number  line  may  perhaps  be 
explained  on  the  basis  of  the  domain  associations.  The  Number-Line 
method  in  its  spatial  approach  uses  a  representation  from  geometry, 
whereas  the  postman  approach  is  heavily  verbal,  and  comes  from  a  farther 
domain*  It  concentrates  more  on  the  concept  and  doesn't  offer  a  break¬ 
down  of  technical  steps  for  solving  the  problems,  as  does  the  other 
method.  Such  an  approach  was  not  very  common  in  the  students'  past 
experience  of  studying  math.  In  our  conventional  schools  the  boundaries 
between  "math"  and  'Stories’1  are  pretty  firm.  It  seems  that  students 
have  internalized  this  arbitrary  distinction  —  they  allocate  separate 
"boxes"  in  their  minds  for  these  ‘different"  domains,  thus  making  it 
difficult  for  the  knowledge  ^nd  experience  which  is  built  into  these 
boxes  to  interact  flexibly. 

Students  who  were  s^jcying  the  Postman  Stories  admitted  that  they 
weren't  using  the  represer  'it ion  presented  in  the  lesson  for  solving  the 
test  problems.  However,  asked  to  resolve  the  problems  using  the 

bills  and  checks,  the^  got  correct  answers  on  those  items  which  they 
had  previously  answered  incorrectly. 

The  following  aie  responses  given  in  two  interviews  with  students 
who  studied  the  Postman  Stories.  They  are  similar  to  to  the  responses  we 
got  from  students  during  the  experiment  and  are  brought  here  in  quite  a 
detail  in  order  to  illustrate  the  point  we  were  making  about  the 
representaion  of  the  postman  stories. 

Case  1.3  The  following  dialogue  took  place  after  the  student 
completed  the  test. 

Is  Now,  when  you  were  doing  the  problems  did  you  ever  think 

about  bills  and  checks  at  all  or  did  you  pretty  much  just.*.? 

S:  I  just  did  it. 

I:  Is  there  any  reason  that  you  can  think  of  why  you  didn't 

think  of  the  bills  and  check? 

Ss  I  don't  know,  I  just  thought  that  I  had  to  do  it,  because 
there  wasn't  time  enough  I  guess,  I  don't  know..* 

Is  OK*  Let's  try  something  like  this,  -4-+5,  using  the  postman 
story. 

S:  Um,  let's  see,  they're  giving  you  a  bill  for  $4.00,  but 
then  they're  taking  away  $5.00  of  a  check* 

Is  OK,  now,  when  you  were  doing  this  before,  how  were  you 
doing  this? 

Ss  I  was  thinkin'  that  you  had  to  change  them  if  they  were 
next  to  each  other. 

Is  OK,  so  the  answer  you  would  have  gotten  sould  be  +9  I  guess, 
right? 

S:  Yeah. 
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Is  Now,  getting  a  bill  for  $4.00  makes  you  what?  What  happens? 

S:  You  lose  $4.00. 

Is  OK,  and  now  what  does  this  piece  here,  the  -+5 ,  mean? 

S:  That  means  that  he's  taking  away  a  check  for  $5.00. 

Is  And  what  happens?  Richer  or  poorer? 

Ss  Poorer. 

I:  OK,  so  what  happens  all  together? 

Ss  Oh,  you're  getting  poorer  all  together,  ok,  so  you're 
losing  $9.00. 

Is  OK,  so  the  answer  is... 

Ss  Negative  nine. 

Is  Why  are  you  losing  $9.00? 

Ss  Because  first  he  gives  you  a  bill  for  $4.00  and  then  he 
takes  away  $5.00  so  you  add  them  altogether. 

Is  OK,  and  so  what's  the  final  answer? 

Ss  Negative  nine. 

Case  2.^  This  dialogue  also  took  place  after  the  student 
completed  the  test  and  the  interviewer  asked  her  to  solve  a  problem  from 
a  task  she  failed  to  answer  correctly  before.  This  time  he  insisted  she 
use  the  postman  story  for  solving  the  problem. 

Is  How  about  this  problems  -2-4? 

Ss  Well,  minus  two  and  take  away  four  and  that  would  give 
you  plus  two. 

I:  Would  you  do  that  for  me  using  the  postman  story? 

Ss  Well,  you  get  a  bill  for  two  dollars,  and  you  get  a  check 
for  four  dollars. 

Is  What  does  "get"  mean? 

Ss  Brings  you,  you  add. 

Is  OK,  so  write  down  what  you  just  said. 

S  s  -2 

4  ,  you  take  away  the  four 

Is  Write  it  horizontally 
Ss  -2-l-H- 

Is  Is  this  the  same  as  -2-4? 

Ss  No,  he  takes  away,  he  takes  away  a  check  for  $4.00.  So 
the  answer  is  minus  six. 

How  are  the  Algorithms  Reflected  in  the  Dimensionality  of  the  Test? 

The  dimensionality  of  achievement  test  data  has  lately  become  a 
major  concern  of  psychometricians  due  to  the  fact  that  for  all  practical 
purposes,  latent  trait  models  are  applicable  only  to  unidimensional 
sets  of  data*  When  such  models  are  applied  to  multi  factorial  sets  of 
data  the  parameter  estimates  are  Inaccurate  if  obtainable  at  all.  (Often 
they  are  not  obtainable  due  to  lack  of  convergence.)  For  example, 
Reckase  (1979)  showed  that  in  data  sets  containing  more  than  one 
independent  factor,  the  three-parameter  logistic  model  picks  one  factor 
and  discriminates  among  ability  levels  on  it,  while  ignoring  the  other 
factors.  On  the  other  hand,  the  one-parameter  logistic  model  seems  to 
estimate  the  sum  of  the  factors.  In  data  sets  which  have  first  factors 


that  are  large  relative  to  the  other  factors  in  the  test,  both  methods 
measure  the  first  factor. 

Studies  on  the  dimensionality  of  achievement  data  in  different 
topics  have  indicated  that  there  are  always  more  than  one  major  factor 
underlying  any  test  data  in  the  achievement  domain  (see  for  example, 
Tatsuoka  &  Birenbaum,  1979;  Gialluca,  1980).  Moreover,  it  has  been 
shown  that  the  dimensionality  of  a  given  test  changes  depending  on  the 
time  point  in  the  instruction  when  it  is  given.  For  example,  Kingsbury 
&  Weiss  (1979)  demonstrated,  using  factor  analytic  techniques,  that  data 
sets  obtained  prior  to  instruction  and  at  the  peak  of  instruction  yield 
different  factorial  structures.  The  variance  accounted  for  by  the  first 
factor  in  their  study  was  considerably  less  at  the  pretest  than  it  was 
at  the  peak  of  instruction. 

It  seems  that  a  plausible  explanation  for  the  raul tidimensionali ty 
in  achievement  data,  especially  of  the  problems  solving  type,  can  be 
provided  by  the  algorithmic  approach.  The  fact  that  students  are  using 
various  rules  of  operation  for  solving  the  test  problems  adds  systematic 
sources  of  variation  in  the  data  resulting  in  an  increase  in  the 
underlying  factorial  structure  of  the  test.  As  for  the  change  in  the 
dimensionality  between  the  pre-  and  the  post-test  —  this  can  be 
explained  as  being  a  result  of  the  fact  that  in  the  post-test  situation, 
students  are  generating  their  rules  of  opeiation  based  on  a  common 
training  or  coaching  treatment  (i.e.,  instruction),  whereas  in  the 
pretest  the  influence  of  this  "common  denominator"  doesn't  exist.  Thus, 
the  variety  of  algorithms  used  in  the  pretest  is  larger,  resulting  in  a 
larger  dimensional  space.  (Needless  to  say,  in  such  situations  where 
the  dimensionality  of  the  test  data  changes  between  pre  and  post 
administrations,  it  becomes  meaningless  to  compute  gain  scores.) 

In  order  to  test  the  effect  of  the  algorithms  on  the  dimensionality 
of  the  test,  we  generated  five  datasets  that  varied  with  respect  to  the 
number  of  algorithms  underlying  the  responses  on  a  set  of  12  tasks. 

Each  dataset  consisted  of  120  cases.  A  quarter  of  the  cases  were 
assigned  random  responses.  Another  quarter  were  assigned  correct 
responses  on  all  12  items.  For  half  of  the  cases  in  the  first  dataset, 
responses  were  generated  to  follow  one  algorithm.  The  number  of 
algorithms  was  increased  by  one  for  each  consecutive  dataset,  thus 
ranging  from  1-5  whereas  the  rest  of  the  responses  were  kept  unchanged 
across  the  five  datasets.  This  simulation  was  meant  to  describe  a 
situation  in  which  25%  of  the  subjects  know  nothing  about  the  topic  that 
is  being  tested,  and  thus  randomly  guess  at  the  answer.  Another  quarter 
of  the  subjects  mastered  the  topic  perfectly  and  got  correct  answers  on 
all  the  items.  Half  of  the  subjects  "mastered"  incorrect  rules,  the 
number  of  which  was  incremented  from  one  dataset  to  the  next.  This 
simulation  was  for  the  purpose  of  examining  the  effect  of  the  increment 
in  the  number  of  algorithms  on  the  dimensionality  of  the  test.  In  order 
to  make  the  effect  of  the  algorithms  clearer,  we  simulated  a  rather 
hypothetical  situation  in  which  75%  of  the  responses  were  perfectly 
reliable,  i.e.,  uncontaminated  by  errors  of  measurement. 
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Table  5  presents  the  results  of  a  principal  components  analysis  for 
each  of  the  five  generated  data  sets.  As  can  be  seen  in  the  table,  the 
magnitude  of  the  first  eigenvalue  decreases  and  those  of  the  second  and 
the  third  eigenvalues  increase  as  the  number  of  algorithms  in  the 
dataset  increases. 

Figure  2  presents  the  scree  test  for  the  five  datasets.  The 
percentages  of  the  common  variance  explained  by  the  first  factor  range 
from  52.1%  in  the  dataset  that  includes  one  wrong  algorithm  to  35.7%  in 
the  dataset  that  includes  five  wrong  algorithms.  Figure  3  illustrates 
the  decrease  in  the  percent  of  variance  explained  by  the  first 
eigenvalues  across  the  five  datasets.  The  reliability  as  estimated  by 
coefficient  alpha  ranges  from  .877  in  the  first  dataset  to  .804  in  the 
fifth  one.  Figure  4  illustrates  the  decrease  in  the  reliability  across 
the  five  datasets. 

We  may  infer  from  this  simulation  that  the  increase  in  the 
number  of  wrong  algorithms  used  by  students  in  solving  the  test 
problems  causes  an  Increase  in  the  underlying  dimensionality  of  the  test. 

In  order  to  test  the  closeness  of  the  simulation  study  results  to 
results  from  real  data,  a  principal  components  analysis  was  done  on  the 
four  datasets  that  were  collected  In  the  January  experiment.  Twelve 
subscores,  each  consisting  of  responses  to  four  parallel  items  were 
computed  for  the  four  datasets.  Table  6  and  Figures  5,  6  and  7  present  the 
results  of  the  principal  component  analysis  on  the  four  datasets.  As 
can  be  seen  in  the  table  and  in  the  figures,  more  than  one  factor 
constitutes  the  underlying  common  space  of  the  test  in  each  of  the  four 
datasets. 

The  first  eigenvalue  in  both  tests  of  the  "Postman1'  group  explains 
a  larger  proportion  of  common  variance  than  do  the  respective 
eigenvalues  in  the  "Number  Line"  data  sets.  The  variety  of  algorithms 
used  in  the  "Postman"  group  was  smaller  than  that  used  in  the  "Number 
Line"  group.  This  further  supports  our  previous  explanation  concerning 
the  effect  of  the  increase  in  the  number  of  algorithms  on  the 
dimensionality  of  the  test.  (See  also  Birenbaum  &  Tatsuoka,  1980.) 
Moreover,  in  both  groups,  the  first  eigenvalue  in  the  second  test 
explains  the  larger  proportion  of  common  variance  than  does  the  first 
eigenvalue  of  the  first  test.  The  first  test  was  given  at  the 
conclusion  of  the  addition  unit,  before  subtraction  was  introduced, 
whereas  the  second  test  was  given  at  the  conclusion  of  the  subtraction 
unit.  Thus,  the  "common  denominator"  of  the  instruction,  which  has  the 
effect  of  reducing  the  variety  of  algorithms,  is  missing  from  the 
responses  to  the  first  test.  As  a  consequence,  the  first  test  has  a 
larger  variety  of  algorithms,  and  this  results  in  a  smaller  amount  of 
common  variance  being  explained  by  the  first  eigenvalue.  As  can  be  seen 
in  Table  6  and  in  Figure  7  the  reliability  coefficients  (ot's)  for  the 
four  tests  are  also  in  the  same  line  with  the  results  discussed  above; 
i.e.,  higher  reliabilities  are  found  for  the  "Postman"  compared  to  the 
respective  tests  in  the  "Number  Line"  group,  and  higher  reliabilities 
accrue  to  the  second  test  compared  to  the  first  in  both  groups. 
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Figure  2.  SCREETEST-  Eigenvalues  extracted  by  a  principal  component  analysis 

for  the  five  simulated  data  sets. 
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Figure  3.  Percent  of  varionce  explained 
by  the  first  eigenvalue  across 
the  five  data  sets. 


Figure  4.  Reliability  coefficients  for 
the  five  data  sets. 


i^envalues  and  percent  of  variance  explained  and 
reliability  coefficients  for  the  four  datasets 
collected  in  t lie  January  experiment 
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Figure  5.  SCREETEST-  Eigenvalues  extracted  by  a  principal  component 
analysis  for  the  five  simulated  data  sets. 
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Summary  and  Conclusions 

The  main  focus  of  this  study  was  on  issues  related  to  the  effect  of 
different  instructional  methods  on  the  consistency  and  types  of 
algorithms  developed  by  students,  as  well  as  on  the  underlying 
dimensionality  of  the  test. 

The  data  used  was  collected  in  an  experimental  setting  in  which  two 
instructional  methods —  the  Postman-Stories  and  the  Number-Line  raethods- 
-  were  used  for  teaching  signed  number  addition  and  subtraction,  157 
seventh  graders  who  had  not  studied  the  topic  before  were  randomly 
assigned  to  each  of  the  experimental  groups,  A  52  item  free  response 
test  consisting  of  13  tasks  of  four  parallel  items  each  was  administered 
twice  during  the  experiment.  The  first  time  —  at  the  conclusion  of  the 
unit  on  addition,  before  subtraction  was  introduced  and  the  second  time, 
at  the  conclusion  of  the  the  unit  on  subtraction.  Eighty-one  students 
from  both  experimental  groups  completed  the  test  twice.  Their  responses 
on  12  of  the  test's  tasks  (those  tasks  which  were  identical  to  the  ones 
used  in  the  previous  study)  constitute  the  four  datasets  used  in  the 
current  study. 

The  two  different  instructional  methods  yield  an  overall 
significant  difference  in  the  rate  of  consistent  responses  in  the  first 
test.  Since  the  two  experimental  groups  were  exposed  to  exactly  the 
same  tasks  throughout  their  entire  learning  experience,  the  significant 
differences  in  the  consistency  rate  between  the  two  groups  seems  to 
imply  differences  in  the  information  processing  mechanisms  involved. 

This  was  especially  noticeable  in  the  first  test,  perhaps  due  to  the 
fact  that  the  test  was  given  before  subtraction  was  introduced.  Thus, 
the  common  denominator  of  the  instruction  has  not  yet  been  affecting  the 
students'  responses  greatly.  This  is  further  supported  in  the  results 
of  the  analysis  concerning  the  underlying  structure  of  the  test  in  the 
four  datasets.  The  dimensionality  of  the  first  test  in  this  analysis 
was  found  to  be  larger  than  that  of  the  second  one  for  both  experimental 
groups. 

The  examination  of  the  types  of  algorithms  developed  by  students 
under  the  two  different  instructional  methods  resulted  in  identifying 
two  categories  of  algorithms.  One  category  referred  to  the  syntactic 
aspect  of  the  problem,  i.e.,  the  structure  of  the  number  sentence.  The 
other  category  referred  to  the  semantic  aspect  —  the  conceptual 
representation  which  that  sentence  reflected.  The  algorithms  were 
classified  into  three  classes;  those  which  were  common  to  both  groups 
and  those  which  were  unique  to  each  experimental  group.  All  the 
algorithms  that  were  common  for  both  groups  and  those  that  were  unique 
to  the  Postman  group  fell  in  the  first  category.  Semantic  algorithms 
were  identified  only  in  the  Number  Line  group.  Interviews  with  students 
who  were  studying  the  Postman  Stories  confirmed  that  they  were  not  using 
the  representation  for  solving  the  test  problems.  However,  when  they 
were  explicitly  asked  to  use  the  representation  after  the  test  was  over, 
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they  successfully  answered  those  test  problems  which  they  failed  to 
answer  correctly  before. 

The  variety  of  algorithms  used  by  students  for  solving  the  test 
problem  was  larger  in  the  Number-Line  group  than  it  was  in  the  Postman 
group.  This  finding  was  further  confirmed  in  a  principal  components 
analysis  that  resulted  in  larger  proportions  of  common  variance 
explained  by  the  first  factor  of  the  Postman  Stories  compared  to  the 
Number  Line  in  both  tests.  This  further  supports  the  assertion 
regarding  differences  in  the  information  processing  mechanisms  developed 
by  students  following  the  different  teaching  methods. 

A  comparison  of  the  consistency  rate  between  the  January  and  the 
November  data  (Birenbaura  &  Tatsuoka,  1980)  showed  a  large  discrepency 
between  the  two.  The  average  consistency  rate  in  the  November  data  was 
much  higher  than  that  in  the  January  datasets.  As  recalled ,  the 
November  data  was  collected  in  the  advanced  stages  of  the  learning 
process  after  students  were  given  sufficient  drill  and  practice  quizzes 
and  tests.  In  contrast,  in  the  January  experiment  many  students  were 
still  in  the  earlier  stages  of  learning  when  the  experiment  was  over. 
These  results  indicate  that  the  point  of  time  in  the  learning  process  in 
which  the  test  is  given  does  make  a  difference  in  terms  of  the 
consistency  of  the  responses.  In  the  early  stages  of  the  learning 
process,  before  the  material  has  been  fully  digested,  students  are  still 
switching  strategies  as  they  are  testing  their  algorithms  and  modifying 
them  whenever  it  becomes  necessary.  The  presentation  of  the  next 
question  in  the  test  sometimes  provides  the  student  with  feedback  as  to 
the  feasibility  of  his/her  algorithm.  It  was  noted  that  in  such 
cases  there  is  a  clear  violation  of  the  local  independence  assunption 
underlying  the  Latent  Trait  models.  Since  local  independence  is  a 
critical  assumption  in  the  ICC  Theory,  when  testing  in  early  stages  of 
the  learning  process  in  which  we  can't  assume  a  stable  latent  trait  (but 
rather  latent  states  which  are  likely  to  change  during  the  test), 
applying  the  ICC  models  would  be  Inappropriate.  In  the  more  advanced 
stages  of  learning,  when  the  algorithms  have  been  stabilized,  the  local 
independence  assumption  is  satisfied  but  one  is  still  faced  with  the 
fact  that  the  data  is  multidimensional,  due  to  the  existence  of 
different  types  of  algorithms.  Thus,  again  practical  Latent  Trait 
models  that  assume  unidimensionality  are  inappropriate.  (Note  that 
while  unidimensionality  is  a  sufficient  condition  for  local 
Independence,  assuming  normality,  local  independence  is  only  a  necessary 
but  not  a  sufficient  condition  for  unidimensionality).  Thus,  the 
development  of  practical  multidimensional  Latent  Trait  models  becomes  a 
necessity  for  this  kind  of  achievement  data. 
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Footnotes 


1)  For  the  summary  of  the  verbal  responses  concerning  the  subtraction 
problems,  see  Appendix  4*  Support  for  this  comes  from  protocols  of 
clinical  interviews  with  students  solving  the  same  version  of  signed- 
number  tests.  Those  protocols  were  kindly  provided  to  us  by  Mr.  Seth 
Chaiklin  of  the  Learning  &  Development  Center  at  the  University  of 
Pittsburgh,  to  whom  we  are  greatly  indebted. 

2)  Note  how  many  correct  answer  this  wrong  algorithm  yields. 

3)  Taken  from:  Summary  of  a  protocal  on  the  Postman  Stories,  Feb.  2,  1980. 
Interviewer:  Seth  Chaiklin.  Summarizers:  Nancy  Strandmark  and 

Steve  Wise. 

4)  Taken  from:  Summary  of  a  protocal  on  the  Postman  Stories  Feb.  1,  1980. 
Interviewer  and  Summarizer:  Seth  Chaiklin. 
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Appendix  1 


The  Singed- Number  Test  Used  in  the  January  Experiment 


Test  Items 


I 

II 

III 

IV 

1. 

-6-(-3)=2 

17. 

-l-(-10)=9 

33. 

—3— (—5 ) =2 

49. 

-2-(-l 1 )=9 

2. 

-7-9—16 

18. 

-2-11—13 

3A. 

-A-6— 10 

50. 

-5-14  —  19 

3. 

1 2H — 3=9 

19. 

7+-5=2 

35. 

154—6=9 

51. 

4+-2=2 

4. 

l-(-10)=l 1 

20. 

3-(-12)=15 

36. 

5-(-7)=12 

52. 

6-(-8)=14 

5. 

-3+12=9 

21. 

-1+10=9 

37. 

—4+1 3=9 

53. 

-2+11=9 

7. 

8-6=2 

23. 

7-3=2 

39. 

4-2=2 

55. 

9-7=2 

8. 

— 16— (—7) =— 9 

2A . 

-12-(-10)=- 2 

AO. 

-ll-C-2)— 9 

56. 

-7-(-5)=- 2 

10. 

-14+-5— 19 

26. 

-10+-1  — 11 

A2 . 

-7+-5— 12 

53. 

-10+-3— 18 

11. 

3+— 5=-2 

27. 

24-11—9 

A3. 

6+— 8— 2 

59. 

1+-10— 9 

12. 

1 3— (  — 4 ) =1 7 

28. 

0-(-9)«9 

A4. 

6-(-4)=I0 

60. 

0-(-2)=2 

13. 

-3— +12=-15 

29. 

-2-+11— 13 

A5. 

-7-+9— 16 

61. 

-4-+6  — 10 

15. 

-6+4—2 

31. 

-5+3—2 

A7 . 

-4+2—2 

63. 

-3+6—2 

16. 

2-11—9 

32. 

5-14—9 

A8. 

7-16—9 

64. 

4-13—9 
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Appendix  4 

Summary  of  a  student's  Verbal  Responses  Concerning  the  Subtraction 
Problems,  Given  in  an  Interview. 

-S-(-L) 

1  -6-(-8)— 14 

Because  they're  both  negatives  and  so  you  just  add 
them  together. 

17  -l-(-10)= 

negative,  no  wait  a  minute,  ...  positive  11  'cause  you 
change  the  signs  ...  cause  they're  all  the  same  .  •» 
because  they're  all  negatives. 

Is  so  you  change  all  of  the  signs?  S:  I  think  so, 

I  don't  really  know. 

33  -3-(-5)«8 

'cause  you  change  all  of  the  signs. 

49  -2-(-ll)»13 

'cause  you  have  to  change  all  of  the  signs  'cause 
they're  all  negatives. 

-S-L 

2  -7-9-2 

oh,  I  was  wrong,  sorry  (later:  because  the  nine  was 
positive  and  it  was  larger) 

18  -2-11-9 

You  subtract  2  from  11  'cause  11  Is  larger. 


I 
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34  -4-6=4-10 

I  guess  because  there  are  3  negative  signs. 

50  -5-14=19 

'cause  you're  changing  the  signs  'cause  they're  the  same. 

S-(-L) 

1 

4  l-( -10)=-9 

'cause  you  add  one,  no  you  subtract  one  from  the  10, 

'cause  10  is  the  larger  number  it  is  negative.  j 

20  3-(-12)  =-9  ! 

: 

'cause  12-3=9  and  12  is  larger  so  you  go  by  negatives.  : 

36  5— ( — 7 ) =1 2  ; 

'cause  there  is  2  negative  signs. 

52  6-(-8) 

S:  -2.  Is  ok.  Tell  me  why.  S:  No,  not  -2,  its 
+14.  I:  why  did  you  think  -2  at  first?  S:  'cause 

I  was  just,  I  didn't  keep  track  of  if  there  were  2  | 

right  next  to  each  other.  Because  2  right  next  to  j 

I 

each  other  canges  the  signs. 

L-S 

7  8-6-14 

They're  both  positive  so  you  just  add  them  regularly. 

23  7-5-2 

'Cause  its  a  regular  ...  two  positive  numbers. 

39  4-2-2 

'Cause  they're  both  the  same  so  you  add  regularly  two 
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positive  number. 

55  9-7*2 

'Cause  they're  both  the  same,  positive. 

-L-(-S) 

8  -16-(-7)=-23 

Since  they're  both  the  same  process  you  just  add 
regularly  equals.  I:  what's  both  the  same?  S:  You're 
just  adding. .. they  have  the  same  signs  so  you  add  them 
regularly. 

24  -12-(-10)=22 

I  am  going  to  say  +22  because  I'm  going  to  change  the 
signs  around.  Is  OK,  and  why  is  that?  S:  because 
they're  all  the  same  sign. 

40  -ll-(-2)=13 

Because  there  are  3  negative  signs  so  you  put  them  over  to 
the  positive. 

56  -7-(-5)=*12 

'Cause  they  are  all  the  same  sign,  all  negatives. 

L-(-S) 

12  13-(-4)«ll 

because  the  13  is  positive  and  has  the  larger  number  than 
the  negative  —  go  to  positives. 

28  0-(-9)— 9 

-9  because  0  is  just,  it  Just  means  that. . .that's  —  now 
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let's  see.  Is  So  how  did  you  get  -9?  S:  Oh,  no  its  +9 
Is  oh,  why?  S:  because  there  are  2  negative  signs 
next  to  each  other 
44  6-(-4)=10 

44  because  you  change  the  signs  ^cause  they're  right  next  to 
each  other. 

60  0“(-2)=2 

'cause  there's  2  negative  signs  right  next  to  each  other. 

-S-+L 

13  -3-+12=15 

let's  see...  I  changed  it  into  positive  3  minus  positive  12. 
No,... Oh,  positive  3  plus  positive  12.  Is  So  positive  15. 

Ss  Yeah,  I  don't  know  if  that's  right. 

29  -2-+11-13 

Is  Tell  me  why.  Ss  'cause  you  change  the  negative  signs 
into  positive  signs  and  then  you  just  add  regularly.  Is  Why 
do  you  change  the  negative  into  positive  signs?  Ss  'cause  well, 
let's  see,  'cause  there  is  2  negative  signs  right  next  to 
each  other. 

45  -7-+9-16 

'cause  you  change  the  2  negative  into  positive  signs. 

61  -4-+6-10 

'cause  there's  2  negative  signs  right  next  to  each  other. 


S-L 


16  2-11-13 


'cause  it's  just  a  regular  problem.  Is  What's  a  regular  problem? 
S:  addition  of  two  positive  numbers. 


32  5-14—9 

'cause  you  can't  subtract  14  from  5  so  you  go  into  negatives. 
48  7-16—9 

'cause  you  cant  subtract  16  from  7  so  you  have  to  go  into 
the  negatives. 

64  4-13—9 

'cause  you  can't  subtract  13  from  4. 


Distribution  List 


Navy 


1  Meryl  S.  Baker 
NPRDC 
Code  P309 

San  Diego,  CA  92152 

1  Dr.  Jack  R.  Borsting 
Provost  &  Academic  Dean 
U.S.  Naval  Postgraduate  School 
Monterey,  CA  93940 

1  Dr.  Robert  Breaux 
Code  N-711 
NAVTRAEQUIPCEN 
Orlando,  FL  32813 

1  Chief  of  Naval  Education  and  Training 
Liason  Office 

Air  Force  Human  Resource  Laboratory 
Flying  Training  Division 
WILLIAMS  AFB,  AZ  85224 

1  CDR  Mike  Curran 

Office  of  Naval  Research 
800  N.  Quincy  St. 

Code  270 

Arlington,  VA  22217 

1  DR.  PAT  FEDERICO 

NAVY  PERSONNEL  RAD  CENTER 
SAN  DIEGO,  CA  92152 

1  Mr.  Paul  Foley 

Navy  Personnel  RAD  Center 
San  Diego,  CA  92152 

1  Dr  .  John  Ford 

Navy  Personnel  RAD  Center 
San  Diego,  CA  92152 

1  Dr.  Norman  J.  Kerr 

Chief  of  Naval  Technical  Training 
Naval  Air  Station  Memphis  (75) 
Millington,  TN  38054 

1  Dr.  William  L.  Maloy 

Principal  Civilian  Advisor  for 
Education  and  Training 
Naval  Training  Command ,  Code  00A 
Pensacola,  FL  32508 


Navy 


1  CAPT  Richard  L.  Martin,  VJSN 
Prospective  Commanding  Officer 
USS  Carl  Vinson  (CVN-70) 

Newport  News  Shipbuilding  and  Drydock  Co 
Newport  News ,  VA  23607 

1  Dr.  James  McBride 

Navy  Personnel  RAD  Center 
San  Diego,  CA  92152 

1  Dr  William  Montague 

Navy  Personnel  RAD  Center 
San  Diego,  CA  92152 

1  Ted  M.  I.  Yellen 

Technical  Information  Office,  Code  201 
NAVY  PERSONNEL  RAD  CENTER 
SAN  DIEGO,  CA  92152 

1  Library,  Code  P2Q1L 

Navy  Personnel  RAD  Center 
San  Diego,  CA  92152 

6  Commanding  Officer 

Naval  Research  Laboratory 
Code  2627 

Washington,  DC  20390 

1  Psychologist 

ONR  Branch  Office 
Bldg  114,  Section  D 
666  Summer  Street 
Boston,  MA  02210 

1  Psychologist 
ONR  West 

1030  East  Green  St. 

Pasadena,  CA  91106 

1  Office  of  Naval  Research 
(442  PT) 

800  N.  Quincy  Street 
Arlington,  VA  22217 

5  Personnel  A  Training  Research  Programs 
‘  (442  PT) 

Office  of  Naval  Research 
Arlington,  VA  22217 


Navy 


Navy 


1  Psychologist 

ONR  Branch  Office 
1030  East  Green  Street 
Pasadena,  CA  91101 

1  Office  of  the  Chief  of  Naval  Operations 
Research  Development  &  Studies  Branch 
(OP-115) 

Washington,  DC  20350 

1  LT  Frank  C.  Petho,  MSC,  USN  (Ph.D) 

Selection  and  Training  Research  Division 
Human  Performance  Sciences  Dept. 

Naval  Aerospace  Medical  Research  Laborat 
Pensacola,  FL  32508 

1  Dr.  Bernard  Rimland  (03B) 

Navy  Personnel  RAD  Center 
San  Diego,  CA  92152 

1  Dr .  Worth  Scanland ,  Director 

Research,  Development,  Test  A  Evaluation 
N-5 

Naval  Education  and  Training  Command 
NAS,  Pensacola,  FL  32508 

1  Dr.  Robert  G.  Smith 

Office  of  Chief  of  Naval  Operations 
OP-987H 

Washington,  DC  20350 

1  Dr.  Alfred  F.  Smode 

Training  Analysis  A  Evaluation  Group 
(TAEG) 

Dept,  of  the  Navy 
Orlando,  FL  3281  3 

1  Dr.  Richard  Sorensen 

Navy  Personnel  RAD  Center 
San  Diego,  CA  92152 

1  Dr.  Ronald  Weitzman 
Code  54  WZ 

Department  of  Administrative  Sciences 
U.  S.  Naval  Postgraduate  School 
Monterey,  CA  93940 

1  Mr  John  H.  Wolfe 
Code  P310 

U.  S.  Navy  Personnel  Research  and 
Development  Center 
San  Diego,  CA  92152 


1  Dr.  Robert  Wisher 
Code  309 

Navy  Personnel  RAD  Center 
San  Diego,  CA  92152 

1  DR.  MARTIN  F.  WISKOFF 

NAVY  PERSONNEL  RA  D  CENTER 
SAN  DIEGO.  CA  Q215? 

Army 


1  Technical  Director 

U.  S.  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

1  Dr.  Beatrice  J.  Farr 

U.  S.  Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

1  Dr.  Myron  Fischl 

U.S.  Army  Research  Institute  for  the 
Social  and  Behavioral  Sciences 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

1  Dr.  Dexter  Fletcher 

U.S.  Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria.VA  22333 

1  Dr.  Michael  Kaplan 

U.S.  ARMY  RESEARCH  INSTITUTE 
5001  EISENHOWER  AVENUE 
ALEXANDRIA,  VA  22333 

1  Dr.  Milton  S.  Katz 

Training  Technical  Area 
U.S.  Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

1  Dr.  Harold  F.  O'Neil,  Jr. 

Attn:  PERI-OK 
Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

1  Mr.  Robert  Ross 

U.S.  Army  Research  Inst*  «ite  for  the 
Social  and  Behavioral  Sciences 
5001  Elsenhower  Avenue 
Alexandria,  VA  22333 


Air  Force 


Marines 


Air  Force  Human  Resources  Lab 
AFHRL/MPD 

Brooks  AFB,  TX  78235 

U.S.  Air  Force  Office  of  Scientific 
Research 

Life  Sciences  Directorate,  NL 
Bolling  Air  Force  Base 
Washington,  DC  20332 

Dr.  Earl  A.  Alluisi 
HQ,  AFHRL  (AFSC) 

Brooks  AFB,  TX  78235 

Dr.  Genevieve  Haddad 
Program  Manager 
Life  Sciences  Directorate 
AFOSR 

Bolling  AFB,  DC  20332 

Research  and  Measurment  Division 
Research  Branch,  AFMPC/MPCYPR 
Randolph  AFB,  TX  781 48 

Dr.  Malcolm  Ree 
AFHRL/MP 

Brooks  AFB,  TX  78235 

Dr.  Marty  Rockway 
Technical  Director 
AFHRL(OT) 

Williams  AFB,  AZ  58224 


Army 


Dr.  Robert  Sasmor 

U.  S.  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

Dr.  Frederick  Steinheiser 

Dept,  of  Navy 

Chief  of  Naval  Operations 

OP-113 

Washington,  DC  20350 

Dr .  Joseph  Ward 
U.S.  Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 


1  H.  William  Greenup 

Education  Advisor  (E03D 
Education  Center,  MCDEC 
Quantico,  VA  22134 

1  Major  Howard  Langdon 

Headquarters,  Marine  Corps 
0TTI  31 

Arlington  Annex 

Golunbla  Pike  at  Arlington  Ridge  Rd. 
Arlington,  VA  20380 

1  Director,  Office  of  Manpower  Utilization 
HQ,  Marine  Corps  (MPU) 

BCB,  Bldg.  2009 
Quantico,  VA  22134 

1  Headquarters,  U.  S.  Marine  Corps 
Code  MPI-20 
Washington,  DC  20 380 

1  Special  Assistant  for  Marine 
Corps  Matters 
Code  100M 

Office  of  Naval  Research 
800  N.  Quincy  St. 

Arlington,  VA  22217 

1  Major  Michael  L.  Patrow,  USMC 
Headquarters,  Marine  Corps 
(Code  MPI-20) 

Washington,  DC  20380 

1  DR.  A.L.  SLAFKOSKY 

SCIENTIFIC  ADVISOR  (CODE  RD-1) 

HQ,  U.S.  MARINE  CORPS 
WASHINGTON,  DC  20380 


CoastGuard 


1  Chief,  Psychological  Reserch  Branch 
U.  S.  Coast  Guard  (G-P-1/2/TP42) 
Washington,  DC  20593 

1  Dr.  Thomas  A.  Warm 

U.S.  Coast  Guard  Institute 
P.O.  Substation  18 
Oklahoma  City,  0U  73169 


Civil  Govt 


Non  Govt 


1  Dr  .  Susan  Chi  pm  an 

Learning  and  Development 
National  Institute  of  Educatio 
1200  19th  Street  NW 
Washington,  DC  202 08 

1  Dr.  Andrew  R.  Molnar 
Science  Education  Dev. 
and  Research 

National  Science  Foundation 
Washington,  DC  20550 

1  Dr .  Joseph  Psotka 

National  Institute  of  Education 
1200  19th  St.  NW 
Washington, DC  20208 

1  Dr .  Vern  W.  Urry 

Personnel  R&D  Center 
Office  of  Personnel  Management 
1900  E  Street  NW 
Washington,  DC  20415 

1  Dr.  Joseph  L.  Young,  Director 
Memory  A  Cognitive  Processes 
National  Science  Foundation 
Washington,  DC  20550 

Other  DoD 


12  Defense  Technical  Information  Center 
Cameron  Station ,  Bldg  5 
Alexandria,  VA  22314 
Attn:  TC 

1  Dr.  William  Graham 
Testing  Directorate 
MEPC0M/MEPCT-P 
Ft.  Sheridan,  IL  60037 


1  Dr .  Wayne  Sellman 

Office  of  the  Assistant  Secretary 
of  Defense  (UFA  A  L) 

2B269  The  Pentagon 
Washington.  DC  20301 

1  DARPA 

1400  Wilson  Blvd. 

Arlington,  VA  22209 


1  Dr. James  Algina 

University  of  Florida 
Gainsville,  Fl  32611 

1  Dr.  Erling  B.  Andersen 
Department  of  Statistics 
Studiestraede  6 
1455  Copenhagen 
DENMARK 

1  Dr .  John  Annett 

Department  of  Psychology 
University  of  Warwick 
Coventry  CV4  7AL 
ENGLAND 

1  1  psychological  research  unit 

Dept,  of  Defense  (Army  Office) 
Campbell  Park  Offices 
Canberra  ACT  2600,  Australia 

1  Dr.  Isaac  Bejar 

Educational  Testing  Service 
Princeton,  NJ  08450 

1  Dr.  Werner  Birke 

DezWPs  im  Streitkraefteamt 
Postfach  20  50  03 
D-5300  Bonn  2 
WEST  GERMANY 

1  Liaison  Scientists 

Office  of  Naval  Research, 

Branch  Office  ,  London 
Box  39  FPO  New  York  09510 

1  Dr.  Robert  Brennan 

American  College  Testing  Programs 

P.  0.  Box  168 

Iowa  City,  IA  52240 

1  DR.  C.  VICTOR  BUNDER SON 
WICAT  INC. 

UNIVERSITY  PLAZA.  SUITE  10 
1160  SO.  STATE  ST. 

OREM,  UT  84057 

1  Dr.  John  B.  Carroll 
Psychometric  Lab 
Unlv.  of  No.  Carolina 
Davie  Hall  013A 
Chapel  Hill,  NC  27514 

l  Dr.  Deborah  Coates 
Catholic  University 
620  Michigan  Ave.  NE 
Washington  D.C.  20064 


1  Military  Assistant  for  Training  and 
Personnel  Technology 

Office  of  the  Under  Secretary  of  Defense 
for  Research  &  Engineering 
Room  30 129,  The  Pentagon 
Washington,  DC  20301 


Dr.  Menucna  Blrenbaum 
School  o£  Education 
Tel  Aviv  University 
Ramat  Aviv  P.0.  Box  39040 
Tel  Aviv  69978 
Israel 

Dr.  Norman  Cliff 
Dept,  of  Psychology 
Univ.  of  So.  California 
University  Park 
Los  Angeles,  CA  90007 

Dr.  William  E.  Coffman 
Director,  Iowa  Testing  Programs 
339  Lindquist  Center 
University  of  Iowa 
Iowa  City,  IA  52242 

Dr.  Allan  M.  Collins 
Bolt  Beranek  A  Newnan,  Inc. 

50  Moulton  Street 
Cambridge,  Ma  02138 

Dr.  Meredith  P.  Crawford 
American  Psychological  Association 
1200  17th  Street,  N.W. 

Washington,  DC  20036 

Dr., Fritz  Drasgow 
Yale  School  of  Organization  and  Manageme 
Yale  University 
Box  1A 

New  Haven.  CT  065 20 

Dr .  Leonard  Feldt 
Lindquist  Center  for  Measurment 
University  of  Iowa 
Iowa  City,  IA  52242 

Dr .  Richard  L.  Ferguson 

The  American  College  Testing  Program 

P.0.  Box  168 

Iowa  City,  IA  52240 

Dr.  Victor  Fields 
Dept,  of  Psychology 
Montgomery  College 
Rockville,  MD  20850 

Univ.  Prof.  Dr.  Gerhard  Fischer 
Lieblggasse  5/3 
A  1010  Vienna 
AUSTRIA 

Dr.  Delwyn  llarnisch 
1CBD 

University  of  Illinois 

51  Gerty  Drive 
Champaign,  II  61801 


1  Dr.  Paul  A.  Games 
403D  Carpenter 
University  Park,  PA  16802 


1  Professor  Donald  Fitzgerald 
University  of  New  England 
Armidale,  New  South  Wales  2351 
AUSTRALIA 

1  Dr.  Edwin  A.  Fleishman 

Advanced  Research  Resources  Organ. 
Suite  900 

4330  East  West  Highway 
Washington,  DC  20014 

1  DR.  ROBERT  GLASER 
LRDC 

UNIVERSITY  OF  PITTSBURGH 
3939  O'HARA  STREET 
PITTSBURGH,  PA  15213 

1  Dr.  Bert  Green 

Johns  Hopkins  University 
Department  of  Psychology 
Charles  &  34th  Street 
Baltimore,  MD  21218 

1  Dr .  Ron  Hambleton 
School  of  Education 
University  of  Massechusetts 
Amherst,  MA  01002 

1  Dr.  Chester  Harris 
School  of  Education 
University  of  California 
Santa  Barbara,  CA  93106 

1  Dr .  Lloyd  Humphreys 

Department  of  Psychology 
University  of  Illinois 
Champaign,  IL  61820 

1  Library 

Hum RR0/ We stern  Division 
27857  Berwick  Drive 
Carmel,  CA  93921 

1  Dr.  Steven  Hunka 

Department  of  Education 
University  of  Alberta 
Edmonton,  Alberta 
CANADA 

1  Dr,  Paul  Holtzman 
Decision  Systems 
MFI 

100  S.  Wacker  Drive 
Chicago,  IL  60606 


1  Dr.  Walter  Bogan 

4615  N.  Park.  Ave,  apt.  1611 
Chevy  Chase,  MD  20015 

1  Dr.  Earl  Hunt 

Dept,  of  Psychology 
University  of  Washington 
Seattle,  WA  98105 

1  Dr.  Jack  Hunter 
2122  Coolidge  St. 

Lansing,  MI  48906 

1  Dr .  Huynh  Huynh 

College  of  Education 
University  of  South  Carolina 
Columbia,  SC  29208 

1  Professor  John  A.  Keats 
University  of  Newcastle 
AUSTRALIA  2308 

1  Mr.  Marlin  Kroger 
1 1 17  Via  Goleta 

Palos  Verdes  Estates,  CA  90274 

1  Dr.  Michael  Levine 

Department  of  Educational  Psychology 
210  Education  Bldg. 

University  of  Illinois 
Champaign,  IL  61801 

1  Dr.  Charles  Lewis 

Faculteit  Sociale  Wetenschappen 
Ri jksuniversiteit  Groningen 
Oude  Boter ingestraat  23 
9712GC  Groningen 
Netherlands 

1  Dr.  Robert  Linn 

College  of  Education 
University  of  Illinois 
Urbana,  IL  61801 

1  Dr.  Frederick  M.  Lord 

Educational  Testing  Service 
Princeton,  NJ  08540 

1  Dr .  Gary  Marco 

Educational  Testing  Service 
Princeton,  NJ  08450 

1  Dr.  David  McArthur 
CSE  145  MOOR  Hall 
UCLA 

Los  Angelos,  CA  90024 


1  Harry  McMahon 

Miami-Dade  Community  College 
District  Administration 
11011  S.W.  104  Street 
Miami,  Florida  33176 

1  Dr.  Scott  Maxwell 

Department  of  Psychology 
University  of  Houston 
Houston,  TX  77004 

1  Dr.  Samuel  T.  Mayo 

Loyola  University  of  Chicago 
820  North  Michigan  Avenue 
Chicago,  IL  60611 

1  Dr.  Melvin  R.  Novick 

356  Lindquist  Center  for  Measurment 
University  of  Iowa 
Iowa  City,  IA  52242 

1  Dr.  Jesse  Orlansky 

Institute  for  Defense  Analyses 
400  Army  Navy  Drive 
Arlington,  VA  22202 

1  Wayne  M.  Patience 

American  Council  on  Education 
GED  Testing  Service,  Suite  20 
One  Dupont  Cirle,  NW 
Washington,  DC  20036 

1  Dr.  James  A.  Paulson 

Portland  State  University 
P.0.  Box  751 
Portland,  OR  97207 

1  MR.  LUIGI  PETRULL0 

2431  N.  EDGEWOOD  STREET 
ARLINGTON,  VA  22207 

1  DR.  DIANE  M.  RAMSEY-KLEE 

R-K  RESEARCH  &  SYSTEM  DESIGN 
3947  RIDGEMONT  DRIVE 
MALIBU,  CA  90265 

1  M INRAT  M.  L.  RAUCH 
P  II  4 

BUN DESM IN ISTER IUM  DER  VERTEIDIGUNG 

P0STFACH  1328 

D-53  BONN  1,  GERMANY 

1  Dr.  David  Miller 

Graduate  School  of  Education 
UCLA 

Los  Angeles,  CA  90024 

l  Dr.  Anthony  J.  Nitko 
School  of  Education 
Division  of  Educational  Studies 
University  of  Pittsburgh 
5C03  Forbes  Quandrangle 
Pittsburgh,  PA  15260 


1  Dr .  Mark  D.  Reckase 

Educational  Psychology  Dept. 

University  of  Missouri-Columbia 
4  Hill  Hall 
Colunbia ,  MO  65211 

1  Dr.  Lauren  Resnick 
LRDC 

University  of  Pittsburgh 
3939  O'Hara  Street 
Pittsburgh,  PA  15213 

1  Dr.  Leonard  L.  Rosenbaum,  Chairman 
Department  of  Psychology 
Montgomery  College 
Rockville,  MD  20850 

1 

1  Dr.  Ernst  Z.  Rothkopf 

Bell  Laboratories 
600  Mountain  Avenue 
Murray  Hill,  NJ  07974 

1  Dr.  Lawrence  Rudner  1 

403  Elm  Avenue 
Takoma  Park,  MD  20012 

1  Dr.  J.  Ryan 

Department  of  Education 

University  of  South  Carolina 

Columbia,  SC  29208  1 

1  PROF.  FUMIK0  SAM EJ IMA 

DEPT.  OF  PSYCHOLOGY 
UNIVERSITY  OF  TENNESSEE 
KNOXVILLE,  TN  37916  1 

1  DR.  ROBERT  J.  SEIDEL 

INSTRUCTIONAL  TECHNOLOGY  GROUP 
HUMRRO 

300  N.  WASHINGTON  ST.  1 

ALEXANDRIA.  VA  22314 

1  Dr.  Kazuo  Shigemasu 

University  of  Tohoku 

Department  of  Educational  Psychology  1 

Kawauchi,  Sendai  980 

JAPAN 

X  Paul  Holtznan 

Decision  Systems  1 

J1FI 

100  S.  Wacker  Drive 
Calcago,  IL  60606 

i  Evelyn  Doody  1 

P.0.  Box  1913 
Seaside,  CA  03955 

1  Dr.  Drew  Malizio 

American  Council  on  Education  1 

No.  1  Pont  Circle,  #20 
Washington  D.C.  20036 


1  Dr.  Edwin  Shlrkey 

Department  of  Psychology 
University  of  Central  Florida 
Orlando,  FL  32816 

1  Dr .  Richard  Snow 
School  of  Education 
Stanford  University 
Stanford,  CA  94305 

1  Dr.  Robert  Sternberg 
Dept,  of  Psychology 
Yale  University 
Box  1 1 A .  Yale  Station 
New  Haven,  CT  06520 

DR.  PATRICK  SUPPES 

INSTITUTE  FOR  MATHEMATICAL  STUDIES  IN 
THE  SOCIAL  SCIENCES 
STANFORD  UNIVERSITY 
STANFORD,  CA  94305 

Dr.  Hariharan  Swaminathan 
Laboratory  of  Psychometric  and 
Evaluation  Research 
School  of  Education 
University  of  Massachusetts 
Amherst,  MA  01003 

Dr .  Brad  Sympson 
Psychometric  Research  Group 
Educational  Testing  Service 
Princeton,  NJ  08541 

Dr.  David  Thlssen 
Department  of  Psychology 
University  of  Kansas 
Lawrence,  KS  66044 

Dr .  Robert  Tsutakawa 
Department  of  Statistics 
University  of  Missouri 
Colunbia,  MO  65201 

Dr.  Howard  Wainer 
Division  of  Psychological  Studies 
Educational  Testing  Service 
Princeton,  NJ  08540 

DR.  SUSAN  E.  WHITELY 
PSYCHOLOGY  DEPARTMENT 
UNIVERSITY  OF  KANSAS 
LAWRENCE,  KANSAS  66044 

Wolfgang  Wlldgrube 
Streitkr aefteamt 
Box  20  50  03 
D-5300  Bonn  2 

Dr.  Steven  Wise 

Dept,  of  Guidance  AEduc.  Psych. 
Southern  Illinois  University 
Carbondale,  II  62901 


